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Executive Summary 

As the economy increasingly demands a more educat- 
ed workforce, states are looking to their colleges and 
universities to meet that demand for a better educated 
citizenry. Not only are individuals with postsecondary 
education likely to be employed, they are also likely 
to be more engaged in their communities through ac- 
tivities such as voting and volunteering.' In 2012, for 
example, overall unemployment rates for graduates of 
four-year and two-year colleges were 4.5 percent and 
6.2 percent, respectively—far below those for individ- 
uals with a high school diploma and without one, 8.3 
percent and 12.4 percent, respectively.” 


But states face fiscal pressures in areas such as health 
care and pensions that will affect their ability to invest 
in postsecondary education. The result is that ques- 
tions pertaining to the effectiveness and efficiency of 
the resources that states and students direct to postsec- 
ondary institutions and systems will be increasingly 
important for the foreseeable future. 


That reality collides with existing state-level account- 
ability systems, which often focus more on inputs (for 
example, enrollments and the amount of spending 
for education by states and students) than outcomes 
(such as completion and employment). Moreover, the 
data have not been used to guide policy decisions in 


areas such as funding. States are making progress, as 
the examples below suggest, but significant room for 
improvement is evident, particularly when it comes 
to gauging the effectiveness and efficiency of the re- 
sources directed at postsecondary education. 


Governors are asking four fundamental questions 
about the effectiveness and efficiency of investments 
in postsecondary educational institutions and systems 
in their states to develop their policy agendas: 

Effectiveness 

1. Are our postsecondary students learning the 
knowledge and skills needed to be successful 
workers and citizens? 

2. Do our postsecondary graduates get jobs, and 
do those jobs pay wages and salaries that can 
support families? 

Efficiency 

3. How does the number of students who gradu- 
ate from our postsecondary institutions compare 
with the number of students who enroll? 

4. What is the return on our states and our stu- 
dents’ investments in postsecondary education 
in terms of completed degrees and certificates? 


States’ postsecondary leaders can provide answers to 
these four fundamental questions with readily avail- 
able data using several indicators of the effectiveness 


'Sandy Baum and Jennifer Ma, Education Pays: The Benefits of Higher Education for Individuals and Society (Washington, DC: College Board, 
2007), http://www.collegeboard.com/prod_downloads/about/news_info/trends/ed_pays_2007.pdf. 


> Bureau of Labor Statistics, U.S. Department of Labor, “Employment Projections,” Table: Earnings and unemployment rates by educational attain- 
ment, http:/Avww.bls.gov/emp/ep_chart_001.htm; and Bureau of Labor Statistics, U.S. Department of Labor, “Labor Force Statistics from the Cur- 
rent Population Survey,” Table 7: Employment status of the civilian noninstitutional population 25 years and over by education attainment, sex, race, 


and Hispanic or Latino ethnicity, http:/Avww.bls.gov/cps/cpsaat07.htm. 
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and efficiency of postsecondary education invest- 
ments recommended in this NGA paper.’ The recom- 
mended indicators presented here have been vetted by 
practitioners, researchers, and policy experts and are 
currently being used by a number of states, includ- 
ing several that participated in a National Governors 
Association (NGA) policy academy on strengthening 
states’ systems of accountability for postsecondary 
education. 


Although the recommended indicators are not the 
only ones that should be considered when evaluating 
a State’s postsecondary education system, they do pro- 
vide a good top-line view of a system’s effectiveness 
and efficiency. Moreover, they allow for customiza- 
tion to the varying institutional missions of research 
universities, comprehensive universities and commu- 
nity and technical colleges. Other measures, such as 
the Common Completion Metrics developed by NGA 
and Complete College America, offer a more detailed 
look at student movement toward certificates and de- 
grees.* 


As governors’ focus has shifted from how much states 
spend on postsecondary education to whether and how 
state dollars are spent to produce better educated work- 
ers and citizens, the limitations of existing state-level 
accountability systems for postsecondary education 
are becoming increasingly apparent. To facilitate the 
development and analysis of postsecondary education 
policy now and in the coming years, states and their 
postsecondary education systems must review their 
analytical capacity and improve their accountability 
systems. Strong state-level accountability systems for 
postsecondary education include both broader and 
more specific measures and ensure that the measures 
that are used complement and do not conflict with 
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statewide goals and priorities such as broadening ac- 
cess or improving success rates for at-risk students. 


Building Capacity to Analyze 
and Use Data 
In order to effectively pursue the state-level develop- 
ment and analysis of postsecondary policy over the 
coming years, it is important that states and their post- 
secondary systems not only review the level of analyt- 
ical capacity at the state level, but also prune existing 
accountability systems instead of simply adding more 
metrics and reports to already overextended systems. 
To address those issues, governors can: 
¢ Call on state postsecondary leaders, working in 
cooperation with institutional leaders, to per- 
form an audit of existing accountability mea- 
sures, reports, and activities, with an eye toward 
eliminating information that is duplicative, out- 
dated, or not used; 
¢ Emphasize to state postsecondary agency heads 
that data and analysis are essential for budget 
and staffing requests, and review their own staff- 
ing to ensure that there is sufficient postsecond- 
ary expertise; and 
¢ Charge the state systems for postsecondary edu- 
cation and their colleges and universities to be- 
gin working with effectiveness and efficiency 
indicators that will ultimately help identify pol- 
icy options that apply to the state’s current and 
future realities. 


States will continue to face pressure to fund key state 
services as state budgets tighten. Systems of postsec- 
ondary education must be prepared to show how pro- 
ductive colleges and universities are with the resourc- 
es that they have and look for ways to improve their 
effectiveness and efficiency. The four fundamental ef- 


>For additional information on the data that are needed and how they can be applied to derive these metrics, see Travis Reindl and Ryan Reyna, 
Complete to Compete: From Information to Action: Revamping Higher Education Accountability Systems (Washington, DC: National Governors 
Association Center for Best Practices, July 2011), http://www.nga.org/files/live/sites/NGA /files/pdf/1107C2CACTIONGUIDE. PDF. 


‘Ryan Reyna, Common College Completion Metrics (Washington, DC: National Governors Association Center for Best Practices, June 2010), 
http://www.nga.org/files/live/sites/NGA/files/pdf/1007COMMONCOLLEGEMETRICS.PDF. 
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fectiveness and efficiency questions and the related indicators presented in this paper provide a comprehensive ap- 
proach that acknowledges the ongoing shift in postsecondary education policy from a focus on inputs to a broader 
emphasis on outcomes and the relationship between resources and outcomes. Together, they provide governors 
with a powerful tool for determining which state polices will best enable more students to attend college, gain new 
knowledge and skills, graduate on time, and get a job upon completion.> 


Effectiveness of Postsecondary Education: Questions and 
Indicators 


¢ Are our postsecondary students learning the knowledge and skills needed to be successful workers and 
citizens? 
The most important question about the effectiveness of postsecondary education, though also the hardest 
to answer, is whether postsecondary students are learning the knowledge and skills needed to be successful 
workers and citizens. Simply graduating more students without ensuring they have gained the knowledge and 
skills necessary to compete in the economy and contribute to their communities is a waste of scarce resources 
and talent. 


Indicators prior to completion of postsecondary education: 


e LEARNING ASSESSMENT RESULTS—Learning assessment results include results from 
examinations such as the Collegiate Learning Assessment (CLA), the ETS® Proficiency Profile, 
ACT’s Collegiate Assessment of Academic Proficiency (CAAP), and locally developed instruments. 


PORTFOLIO EVALUATION RESULTS—The Association of American Colleges and Uni- 
versities has led a national effort to develop rubrics, or standards of performance, for use in evaluat- 
ing students’ competency in 16 different learning outcome areas as demonstrated in student work 
products collected by faculty or gathered on student electronic portfolios. 


Indicators subsequent to completion of postsecondary education: 


e ALUMNI AND EMPLOYER SURVEY RESULTS—Alumni and employer survey results 
provide helpful context for assessment and employment results and can indicate specific areas for 
improvement. 

States need evidence that postsecondary students are accumulating knowledge and skills not merely 

advancing from one level to the next. Moreover, it is important for states to gauge the quality of learning 

from different perspectives, such as student performance before and after graduation, as well as shorter 
and longer term feedback from employers. 


The focus in analyzing data on these indicators of the effectiveness of postsecondary education should 
be on trends and patterns. Are assessment and placement results improving or declining? Among which 
institutions and/or groups of students? These indicators should be viewed as triggers for more in-depth 
analyses using measures such as those listed below in “Other Measures to Consider.” 


>A new framework for calculating a single higher education productivity ratio for an institution or an educational sector was presented in a 2012 re- 
port written by a panel of the National Research Council with support from Lumina Foundation. This framework represents a potential improvement 
in efforts to gauge how—or whether—different spending and staffing patterns lead to improved productivity in higher education. Although questions 
about the framework’s complexity and utility stand in the way of widespread use and evaluating the framework is beyond the scope of this issue 
brief, the new framework’s development and its reliance on many of the same indicators and measures laid out in this issue brief provide additional 
support for state systems seeking to improve the effectiveness and efficiency of their institutions of higher education. 
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Funding. Missouri’s outcomes-based funding formula for secondary institution includes perfor- 
mance on general education and major-level assessments as an institutional performance metric. 


Public accountability. Massachusetts is developing a plan for measuring learning outcomes 
for postsecondary education that combines several metrics to create composite indicators of student 
learning. 


Academic policy. Students at South Dakota’s public universities are required to pass an as- 
sessment of basic skills before moving from general education to major-level courses. 


INSTITUTION-LEVEL QUALITY AUDIT—An institution-level quality audit complements top- 
line data by identifying the presence or absence of practices that successfully promote learning (such 
as learning communities or first-year learning seminars). 


LICENSURE/CERTIFICATION RATE—Licensure and certification rates offer another form of 
labor market feedback regarding readiness to enter specific occupations and professions. 


ACCEPTANCE RATE FOR GRADUATE STUDIES—This measure rounds out the stu- 
dent outcome data, especially for more selective colleges and universities. 


¢ Do our postsecondary graduates get jobs, and do those jobs pay wages and salaries that can support fami- 
lies? 
Postsecondary education has the potential to contribute to states’ economic development through the prepa- 
ration of individuals for jobs and careers. As a result, it is important that states have reliable and regularly 
updated information on labor market outcomes (i.e., job placement rates and earnings) for the graduates of 
their education and training programs. 


e JOB PLACEMENT RATES FOR POSTSECONDARY GRADUATES (1, 3, 5,10+ years out 
from completion of postsecondary education) 


e EARNINGS FOR POSTSECONDARY GRADUATES (1, 3, 5, 10+ years out from completion 
of postsecondary education) 


Getting a job, though not the only measure of success in postsecondary education, is a key outcome for eco- 
nomic security and full participation in society. Trend data make it clear that without a credential of labor market 
value, an individual's ability to earn a decent living and contribute to their community is becoming increasingly 
difficult. 


Because employment and earnings for many graduates improve significantly in the years following gradua- 
tion, it is important to look at salary and placement data over time, States should also make efforts to track 


graduates’ economic outcomes across state lines as some institutions supply both regional and national labor 
markets. Finally, it is important to exercise caution when looking at data regarding job placement in a graduate’s 
field of study. Connections are harder to make beyond science, technology, engineering, and math (STEM) 
fields, where many graduates may be successfully employed “out of field.” 


Public accountability. States such as Connecticut and Nevada now require postsecondary institu- 
tions to disclosure employment data of recent graduates to the public as a measure in their postsecondary 
accountability systems. 


Funding. Performance funding in Missouri is partially based on outcomes and includes placement of re- 
cent graduates as a performance metric for the state’s technical college. Tennessee’s outcomes-based 
formula for funding community colleges also contains job placement as a performance measure. 


® Jayne E. Brownell and Lynn E. Swaner, Five High-Impact Practices: Research on Learning Outcomes, Completion, and Quality (Washington DC: 
Association of American Colleges and Universities, November 2010); and Colleen Moore and Nancy Shulock, Student Progress toward Degree 
Completion: Lessons from the Research Literature (Sacramento CA: California State University of Sacramento, Institute for Higher Education Lead- 
ership & Policy, 2009), http://www.csus.edu/ihelp/PDFs/R_Student_Progress Toward Degree _Completion.pdf. 
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Efficiency of Postsecondary Education: Questions and Indicators 


¢ How does the number of our students who graduate from our postsecondary institutions compare with the 
number of students who enroll? 
The number of students who enroll in postsecondary education is a bedrock performance measure of col- 
leges’ and universities’ success in improving students’ access to such education. Another bedrock perfor- 
mance measure is the number of postsecondary students who complete their degree or certificate program. 
Comparing these two key measures in relation to each yields an indicator of the efficiency of postsecondary 
education. 


e ANNUAL NUMBER OF POSTSECONDARY DEGREES AND ONE-YEAR AND LON- 
GER CERTIFICATES AWARDED PER 100 FULL-TIME EQUIVALENT POSTSECOND- 
ARY STUDENTS ENROLLED 


The economy's demand for a more educated workforce, combined with limited state resources, raises the stakes 
for getting students into and through certificate and degree programs as efficiently as possible. This indicator 
tracks student completions in relation to access to postsecondary education, and sets the stage for discussions 
about where there may be leaks and bottlenecks in the state's postsecondary pipeline. 


Ideal performance on this indicator would have be having all students who enroll in postsecondary institutions 
complete their degree or certificate on time. At four-year postsecondary institutions, the ideal would be 25 comple- 
tions per 100 full-time equivalent (FTE) enrollments, since one-quarter of all students enrolled would ideally finish 
a bachelor’s program each year (current national performance is 20.8 completions per 100 FTE students). At 
two-year postsecondary institutions, the ideal would be 50 completions per 100 FTE enrollments (current national 
performance is 14.5 completions per 100 FTE students).’ The fact that a significant number of students at two- 
year institutions transfer to other institutions before completion lowers two-year institutions’ performance on this 
indicator. 


Because significant year-to-year change in this indicator is unlikely, states should watch for change in this indica- 
tor over time. Positive change in this indicator may result from a desired improvement in completion. On the 
other hand, such change may result from a decrease in overall enrollment that allows institutions to concentrate 
resources on fewer students or from new policies that lead to a greater selectivity in student admissions, both of 
which may have the unintended consequence of limiting access to postsecondary education. An increase in the 
time taken to complete might also result from an increase in enrollment of students who traditionally need more 
time and greater support in order to complete. To get a full picture, states should track this indicator alongside 
changes in postsecondary enrollment and completion rates and disaggregated by key student characteristics, as 
well as transfer and retention rates. 


Institutional performance contracts. In Colorado, newly formed agreements between the state and 
the public colleges and universities feature a menu of performance metrics, including benchmarking an insti- 
tution’s performance on undergraduate credentials per 100 students enrolled to be within the top 25 percent 
of peer institutions across the nation. 

Funding. Nevada’s outcomes-based funding model includes this indicator as a metric for allocating per- 
formance pool funds. In Tennessee, the outcomes-based funding formula for public four-year institutions 
includes this indicator as a required performance measure. 

Public agenda setting. Connecticut’s newly adopted public policy framework for higher education 
includes this indicator as one of its student success indicators. 


National Center for Higher Education Management Systems (NCHEMS), NCHEMS National Information Center for Higher Education Policy- 
making and Analysis (website), http:/Avww.higheredinfo.org. 
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STUDENT ENROLLMENT—Enrollment serves as a measure of student access to postsecondary 
education and should be disaggregated by type of student. 


STUDENT RETENTION RATE—This measure accounts for attrition within the postsecondary educa- 
tion system; falling retention rates are a leading indication that performance on the recommended indicator is 
declining. 


STUDENT TRANSFER RATE—The student transfer rate provides additional context on the perfor- 
mance of two-year institutions on the recommended indicator, since transfer is an important part of their 
mission, but is not typically reported as a completion. 


TIME TO CREDENTIAL —This measure gauges the pace of student progress through the postsecond- 
ary education system. Reductions in this measure may help improve performance on the recommended 
indicator. 


¢ What is the return on our state’s and our students’ investments in postsecondary education in terms of 
completed degrees and certificates? 
States and students invest more than $130 billion per year in public colleges and universities, with students 
providing an increasing share of that investment.’ As primary investors, both states and students should have 
information about key outcomes, especially completion, in relation to their investment. 


COMPLETIONS OF POSTSECONDARY DEGREES AND ONE-YEAR AND 
LONGER CERTIFICATES PER $100,000 OF STATE APPROPRIATIONS 


COMPLETIONS OF POSTSECONDARY DEGREES AND ONE-YEAR AND 
LONGER CERTIFICATES PER $100,000 IN STUDENT DEBT 


State funding constraints and increasing reliance on tuition are increasing pressure on colleges and universi- 
ties to get the most out of every dollar they receive from taxpayers and students. For that reason, it is important 
for governors to have an indicator that connects payments to public colleges and universities by the state and 
students with outcomes (completed certificates and degrees). 


Because these indicators are not likely to vary significantly from year to year, governors should look for patterns 
over multiple years. Improvement will occur because of increasing completion, declining payments or student 
debt, or a combination of the two. 


Without accounting for transfers to other academic institutions, current national performance on this indicator 
these indicators for public two-year institutions stands at 1.79 completions per $100,000; for public research 
universities it is 1.74 completions per $100,000; and for public comprehensive universities, it is 2.40 completions 
per $100,000.° 

Institutional performance contracts. |n Colorado, agreements recently executed with the state’s 


colleges and universities include degrees awarded per $100,000 in total operating revenues as part of a 
menu of required performance measures. 


Funding. Missouri’s outcomes-based performance funding includes completed credit hours per 
$100,000 of state appropriations as a performance metric for select community colleges. 


Public agenda setting. Connecticut’s newly adopted public policy framework for higher education 
includes a variant of this indicator (state/local appropriations per completion) as one of its affordability and 
sustainability indicators. 


’ State Higher Education Executive Officers, State Higher Education Finance, FY 2012 (Boulder, CO: March 2013), http://www.sheeo.org/sites/de- 
fault/files/publications/SHEF%20FY%2012-20130322rev.pdf. 

National Center for Higher Education Management Systems (NCHEMS), NCHEMS National Information Center for Higher Education Policy- 
making and Analysis (website), http:/www.higheredinfo.org. 


PAGE 6 


NATIONAL GOVERNORS ASSOCIATION 


COMPLETIONS PER $100,000 IN INSTITUTIONAL COSTS ONLY FOR TEACHING 
STUDENTS, EXCLUDING ALL OTHER EXPENDITURES (Education and Related spending 


costs)—Because Education and Related spending is essentially a reflection of institutional/system decision- 
making, this metric is most useful in campus/system management settings. 


Important Cautions for Calculating and Using Performance 
Indicators 

Three important caveats should be heeded by state and postsecondary education leaders when selecting and using 
performance indicators to gauge the effectiveness and efficiency of resources devoted to postsecondary education 
in producing better educated workers and citizens: 


¢ Make sure that postsecondary performance indicators align with and do not conflict with the state’s goals 
or policy priorities. States can boost attainment and efficiency in postsecondary education in a number of 
ways, but some approaches may work at cross-purposes with a state’s goals and priorities. As an example, 
one way for colleges and universities to increase their postsecondary degree attainment rates might be to 
limit the admissions of less prepared students; yet this approach might conflict with the state’s policy objec- 
tive of serving more low-income or first-generation students. To reduce the likelihood of creating perverse 
incentives that would cause institutions to serve fewer students, especially at-risk students, it is important to 
do the following: (a) break down indicators by type of institution and student to ensure that improvements in 
one area are not occurring at the expense of another; and (b) look at related indicators to help determine the 
cause or effect of a particular trend. 


¢ Use acombination of common and mission-specific performance indicators for postsecondary institutions 
and systems. Research universities, comprehensive universities, and community and technical colleges serve 
different types of students and have different products, stakeholders, and funding streams. For that reason, 
accountability systems for postsecondary institutions should include a combination of common and mission- 
specific indicators. For common indicators, such as those recommended here, it is important to calibrate 
performance expectations according to the mission or type of institution. 


¢ Focus on trends in postsecondary performance indicators instead of single-year changes in performance. 
Indicators of the effectiveness and efficiency of postsecondary institutions and systems such as those rec- 
ommended in this paper provide the most useful insight when analyzed over time. Such indicators typically 
do not change dramatically from year to year, and significant year-to-year change in performance could be 
caused by unusual circumstances such as natural disasters, the arrival or departure of a major employer, or 
significant swings in the local, state, or national economy. 


Garrett Groves 

Senior Policy Analyst 
Education Division 

NGA Center for Best Practices 
202-624-5253 


September 2013 
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